@StefanoFioravanzo @elikatsis Is it an idea to merge this pull request into a new branch so others can help with further development?

Update:

Add ability to snapshot all the PVCs atached to the pod. If using snapshot_notebook snapshot all PVCs connected to the notebook and add some notebook metadata as labels (pod name, container name, if the volume was the workspace directory) and annotations (mount path and docker image of the container) to the snapshot. A list_pvc_snapshots function is added that can be used to list snapshots filter using a label selector. For example: list_pvc_snapshots(label_selector="pod=dev-0, is_workspace_dir=True"). Eventually, the plan is to be able to start a new notebook server from a (set of) snapshot(s) using the metadata, similar to how Rok functions.

Update:

I have just added the ability to restore a Notebook server from a "Notebook snapshot". In reality this is an individual snapshot of each PVC mounted to a Notebook server with some metadata written to each snapshot. You can call the function restore_notebook and pass it the snapshot name of one of the PVCs that was mounted to the notebook. The function will then discover other PVCs that were mounted, create new PVCs from the snapshots of those PVCs and finally start a new Notebook server with the new PVCs and the metadata from the snapshots (name, image, namespace).

Update:
With the latest commit I have added functions that check if the provisioner of the PVC has an accompanying VolumeSnapshotClass rather than hardcoding what storage classes are supported. This also checks if a VolumeSnapshotClass actually exists, as you could be using CephFS, CephRDB or another storage provisioner without having added the VolumeSnapshotClass in which case it also isn't possible to take volume snapshots.

This is a first step to allow Kale to be storage provider agnostic when it comes to taking snapshots, as the other functions I wrote should universal.

For this functionality to work the following ClusterRole and ClusterRoleBinding are needed to be able to detect the VolumeSnapshotClasses and Storage Classes. As before, this might not be secure in every environment so proceed with caution.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: view-snapshot_storage_classes
rules:
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list"]

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: view-snapshotclasses-nb-admin
  namespace: admin
subjects:
- kind: ServiceAccount
  name: default-editor
  namespace: admin
roleRef:
  kind: ClusterRole
  name: view-snapshot_storage_classes
  apiGroup: rbac.authorization.k8s.io

@davidspek do I understand correctly that by default Kale can stores snapshots only in Rok storage from Arrikto project?
I'm now fighting with following:

Message: compile_notebook() got an unexpected keyword argument 'auto_snapshot'

I guess I must have this Rok storage as additional component. If MiniKf is free to check Kale locally then maybe should I switch it at the begging ?

And how is going with testing your code from this PR? I just see here the WIP status, so I think it's still in developing, right ?

I have separated Ceph cluster with CephFS and it already connected with K8s with using CSI. So, maybe I can check it with RWX volume from CephFS storage class.

@mr-yaky Kale at the present time only supports Rok. MiniKF includes Rok so you can use that to test with all features enabled.

As you can see I haven't looked at this code in a month now, as I was busy with a KFP PR to automatically set the namespace in notebooks (which has now been merged) along with other work.
The current state of this PR is that all the functions work when running them manually. It also correctly checks what snapshot providers you have and if the PVC belongs to a storage class that has a snapshot class associated with it. At that point you will also be able to use the buttons in the UI, and you can have it create a snapshot when you enable use this notebook's volume. However, when choosing use this notebook's volume it will error on trying to figure out if the snapshot was successfully created. So at this point, it is not yet completely usable.

The next step, once the above error is figured out (and probably a few more), is too look at how to properly implement the metadata. At the moment the snapshot name is hardcoded, thus creating a second snapshot from a notebook will cause an error. This could be solved by adding a UUID, however I was thinking a meaningful UUID such as the snapshot content might be more useful for backtracking a snapshots source later on. Similarly, while it is already possible to restore a notebook from a snapshot of a pvc that was mounted to the notebook (even if it had multiple PVCs) by creating new PVC(s) from the snapshot, there should be a closer look at what all the relevant metadata is that should be used for this.

I've been in contact with @StefanoFioravanzo about this for a while, and I hope to have some feedback on the code and the implementation soon to be able to look at this further. If you have the time and energy to test out this PR and provide feedback and suggestions it is very welcome as this is my first ever attempt at programming. If you have deployed the external-snapshotter to your cluster and have a snapshot class for CephFS setup you should be able to test this with RWX volumes (but seeing as it uses the generic CSI driver it should work for all storage backends that have snapshot support).

Got it, then I'll try to test first with MiniKF and I'll be able to test this PR on next integration step for Kubeflow.
Thank you for the good explanation about the status of PR!